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ABSTRACT 


This  thesis  examines  the  role  of  independent  validation 
in  the  development  of  software  systems.  As  software  systems 
become  increasingly  larger  and  more  complex  the  role  of 
software  validation  becomes  crucial.  In  particular,  one  must 
make  sure  that  the  specification  of  a  software  system  is 
correct  with  respect  to  customer  expectations.  We  introduce 
an  approach  for  developing  and  validating  reuse  libraries  of 
temporal  formal  specifications.  These  libraries  include  UML 
statechart  based  assertions  for  formal  specifications  and 
their  associated  validation  test  scenarios.  We  build  the 
validation  test  scenarios  with  the  goal  of  ensuring  that 
specifications  within  the  libraries  are  indeed  error-free 
and  consistent. 
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INTRODUCTION 


I . 


A.  MOTIVATION 

Software  is  essential  to  almost  every  facet  of  our 
daily  lives  from  business  to  science,  and  while  the 
advantages  are  numerous  and  have  arguably  bettered  our 
lives,  it  comes  with  a  cost.  The  National  Institute  of 
Standard  and  Technology  (NIST)  sponsored  a  study  in  2001  and 
found  that  the  annual  cost  of  software  errors  to  the  U.S. 
Economy  in  2001  was  approximately  $59.5  billion. ^ 
Additionally  the  study  found  that  over  half  the  costs  have 
been  borne  by  the  users.  This  is  remarkable  because  software 
practitioners  have  not  yet  been  held  accountable  to  the  same 
standards  imposed  on  engineers  in  traditional  engineering 
disciplines . 

Over  the  years,  some  software  defects  have  resulted  in 
human  injuries,  property  damage,  and  in  extreme  cases,  loss 
of  human  lives.  This  is  a  cost  that  is  unacceptable  to  users 
and  must  not  be  accepted  by  software  developers.  One  of  the 
most  well  known  examples  of  software  error  causing  human 
fatalities  is  the  THERAC  25^.  This  machine  was  supposed  to 
save  human  lives  by  sending  the  proper  amount  of  radiation 
into  patients,  but  instead  it  overdosed  humans  with  massive 


j 

National  Institute  of  Standards  and  Technology  (NIST) ,  "Software 
errors  cost  U.S.  economy  $59.5  billion  annually."  National  Institute  of 
Standards  and  Technology  (NIST2002-10)  (2002), 

http://www.nist.gov/public_affairs/releases/n02-10.htm  (accessed  May  10, 
2008)  . 

2  N.  Levenson  and  C.  Turner,  "Investigation  of  the  Therac  25 
accidents."  IEEE  Computer  (July  1993),  18-41. 
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amounts  of  radiation  and  resulted  in  several  fatalities.^ 
These  failures  are  not  just  unacceptable  but  could  have  been 
avoided  had  the  software  been  validated  and  verified 
properly.  The  IEEE  standards,  for  validation  and 
verification,  help  to  guide  the  software  developers  with  two 
main  questions,  "am  I  building  the  right  product?"  and  "am  I 
building  the  product  right?"  These  questions  are 
longstanding  and  will,  if  answered  appropriately,  help 
reduce  the  risk  of  mishaps  due  to  software  defects. 

Another  challenge  the  software  industry  faces  is  that 
software  is  increasing  in  complexity,  making  it  difficult  to 
detect  errors  and  eliminate  them.  This  also  increases  the 
importance  of  validation  and  testing  methods  that  enable 
earlier  and  more  effective  error  identification  and  removal. 
Software  must  be  verified  and  validated  to  ensure  not  just 
quality  and  safety  but  also  guard  against  waste,  in  terms  of 
money  and  lives. 

Our  motivation  for  the  research  reported  here  is  to 
develop  techniques  that  improve  the  engineer' s  ability  to 
validate  software  systems.  Additionally,  these  validation 
techniques  will  be  applicable  to  the  entire  software 
industry  and  when  used  in  conjunction  with  verification  will 
hopefully  result  in  better  software  systems  and  reduce 
software  defects  and  their  attendant  costs. 


3  R.  Merritt,  "Embedded  experts:  fix  code  bugs  or  cost  lives." 
Information  Week  (April  10,  2006),  http://www.informationweek.com/news/ 
management/showArticle . jhtml?articleID=185300011  (accessed  May  05, 

2008)  . 
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B. 


INDEPENDENT  VALIDATION  AND  VERIFICATION 


The  current  guideline  for  validation  and  verification 
is  the  IEEE  standard  1012-20044  which  has  had  to  evolve 
because  of  decades  of  unsuccessful  software.  The  overall  aim 
is  to  establish  guidelines  for  the  software  industry  to 
follow  and  help  to  create  better  software  products. 

The  following  is  directly  quoted  from  the  IEEE  standard 
1012-20045: 

Software  V&V  processes  consists  of  the  following: 

•  Verification  process  and  validation  process.  The 

verification  process  provides  objective  evidence 
whether  the  software  and  its  associated  products  and 
processes  conform  to  requirements  (e.g.,  for 
correctness,  completeness,  consistency,  accuracy)  for 
all  life  cycle  activities  during  each  life  cycle 
process  acquisition,  supply,  development,  operation, 
and  maintenance) satisfy  standards,  practices,  and 
conventions  during  life  cycle  processes. 

•  Successfully  complete  each  life  cycle  activity  and 

satisfy  all  the  criteria  for  initiating  succeeding  life 
cycle  activities  (e.g.,  building  the  software 
correctly) . 

•  The  validation  process  provides  evidence  whether  the 

software  and  its  associated  products  and  processes 
satisfy  system  requirements  allocated  to  software  at 

the  end  of  each  life  cycle  activity. 

•  Solve  the  right  problem  (e.g.,  correctly  model  physical 
laws,  implement  business  rules,  use  the  proper  system 
assumptions) . 

•  Satisfy  intended  use  and  user  needs. 


4  Institute  of  Electrical  and  Electronics  Engineers,  Standard  for 
Software  Verification  and  Validation, IEEE-STD-1012,  June  08,  2005. 

5  Institute  of  Electrical  and  Electronics  Engineers,  Standard  for 
Software  Verification  and  Validation, IEEE-STD-1012,  June  08,  2005. 
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The  verification  process  and  the  validation  process  are 
interrelated  and  complementary  processes  that  use  each 
other' s  process  results  to  establish  better  completion 
criteria  and  analysis,  evaluation,  review,  inspection, 
assessment,  and  test  V&V  tasks  for  each  software  life  cycle 
activity . 

The  IEEE  guidelines  leave  the  software  industry  to 
develop  their  own  validation  and  verification  methods.  The 
industry  has  yet  to  integrate  these  techniques  fully, 
contributing  to  the  poor  record  of  software  acquisition.  In 
fact,  the  Standish  Group  reported  that  the  success  rate  of 
software  projects  was  35%  in  2006,  and  the  report  claimed 
that  software  developers  fielded  only  46%  of  the  required 
features  and  functions,  which  means  that  the  projects  did 
not  meet  the  needs  of  the  customer. ^  These  studies  indicate 
conformance  to  the  V&V  guidelines  are  not  enough  to 
significantly  lower  the  defect  rate  in  software  partition  of 
systems.  The  V&V  guidelines  empower  the  individual  to  create 
and  implement  their  own  IV&V  plan  of  actions,  and  the  only 
consequence  to  not  following  the  guidelines  is  unsuccessful 
acquisition  of  software.  Formal  V&V  (FV&V)  techniques  can  be 
used  to  address  some  of  the  failings  of  the  existing 
practice  of  V&V. 

C.  THE  NEED  FOR  A  STANDARD  TECHNIQUE 

The  current  problem  facing  the  software  industry  in 
facilitating  validation  is  that  there  are  no  concise,  simple 
techniques  to  conduct  validation.  The  IEEE  IV&V  guidelines 
are  general  and  meant  to  guide  the  industry  in  the  actual 
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The  Standish  Group  International,  "Annual  Chaos  Report."  (2006)  . 
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implementation  of  validation.  This  leaves  the  software 
industry  to  its  own  devices  on  the  implementation  of 
validation  and  can  result,  in  the  worst  case  scenario,  with 
poor  validation  processes  if  validation  is  conducted  at  all. 
This  is  largely  due  to  the  ignorance  of  validation 
procedures  and  misunderstanding  of  the  necessity  of 
validation.  And  it  has  resulted  in  the  industry's  primary 
focus  on  verification  because  it  can  be  easier  to  accomplish 
and  minimal  effort  is  put  into  validation.  Thus  much  effort 
is  in  "have  we  built  the  system  right?"  but  not  "have  we 
built  the  right  system?" 

Adding  to  the  problem,  there  is  no  general  consensus 
among  the  academic  community  on  how  to  complete  the 
validation  phase,  many  different  paths  are  used.  The  typical 
way  for  a  system  to  be  built,  if  there  is  structure  at  all, 
is  for  the  software  requirements  to  be  gathered,  using  pen 
and  paper,  use  cases  built  and  then  code  is  written  directly 
from  the  use  cases.  There  is  no  formal  link  from  the 
requirements  to  the  formal  specification  of  the  system 
behaviors  (if  one  exists)  to  ensure  that  the  correct  system 
is  being  built.  The  formal  specification  of  system  behaviors 
includes  assertions  which  precisely  model  the  required 
behavior  of  the  system  and  can  be  traceable  to  the  system 
requirements  providing  a  means  to  ensure  that  the  correct 
system  is  being  built.  As  examples,  both  Voyager  and 
Galileo  had  significant  software  errors;  the  primary  cause 
of  the  faults  were  directly  related  to  system  behaviors  that 
had  not  been  identified  or  developed  by  the  developers.  One 
can  significantly  reduce  these  kids  of  errors  by  formally 
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specifying  the  required  behaviors  in  terms  of  assertions  and 
validating  the  correctness  of  these  assertions  against 
stakeholder  expectations  before  building  the  software. 

One  way  to  facilitate  the  validation  process  is  through 
execution-based  validation.  Execution-based  validation  is 
the  process  of  inferring  certain  behavioral  properties  to 
exercise  the  system  under  test  (SUT)  in  a  known  environment 
and  with  selected  inputs.  This  gives  the  person  conducting 
validation  the  capability  to  validate  that  the  system  being 
built  is  the  correct  system  based  on  user  requirements.  In 
our  thesis  we  use  the  StateRover  white-box  automatic  test- 
generator.  The  white-box  test  generator  constructs  a  JUnit 
TestCase  class  from  a  given  statechart  assertion  model  and 
the  associate  embedded  assertions.  The  advantages  of  this 
process  include:  the  ability  to  pinpoint  specific  errors; 
investigate  the  causes  of  failures  on  a  specific  input  in 
detail;  and  eliminate  errors  in  their  design  in  an  efficient 
manner . 

D.  THE  ROLE  OF  SOFTWARE  REUSE 

Software  reuse  is  an  important  concept  that  can  help 
clarify  validation  techniques,  making  them  more  relevant  for 
software  development  teams.  Software  reuse  aims  to  increase 
the  productivity,  efficiency  and  quality  of  software  by 
reusing  the  applicable  software  from  one  project  in  another 
project.^  By  reusing  the  software  the  developers  can  save 
resources  that  would  have  otherwise  been  used  to  develop  the 
software . 

W.  Lim,  Managing  software  reuse,  a  comprehensive  guide  to 
strategically  reengineering  the  organization  for  reusable  components. 
Upper  Saddle  River,  New  Jersey:  Prentice  Hall  PTR,  (1998)  :  7. 
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An  important  part  of  validation  is  the  creation  of 
formal  specifications  in  the  form  of  assertion  statements  to 
capture  the  correct  behavior  of  the  software  from  natural 
language  requirements.  Assertion  statements  can  be  difficult 
to  define  and  produce  because  of  natural  language 
ambiguities.  However,  with  the  use  of  the  reuse  concepts, 
libraries  can  be  established.  These  libraries  will  contain 
correct  assertion  statements  which  have  been  thoroughly 
tested.  The  assertion  development  team  can  then  reuse  the 
correct  assertion  statement  and  use  the  accompanying  test 
suite  to  ensure  that  the  chosen  assertion  matches  the 
requirements  and  proper  validation  is  occurring. 

E .  OUTLINE 

This  work' s  main  objective  is  to  facilitate  the 
assertion  validation  process.  This  is  accomplished  through 
the  use  of  libraries  which  contain  consistent  and  accurate 
assertions.  We  intend  to  demonstrate  that  these  assertions 
are  correct  and  reusable  through  the  use  of  testing 
scenarios.  These  assertions  will  provide  a  type  of 
engineering  control  for  the  IV&V  process. 

The  organization  of  the  thesis  is  as  follows: 

Chapter  II  provides  background  information  about  IV&V 
and  software  reuse.  The  chapter  will  show  a  deficiency  in 
the  current  guidance  provided  for  software  validation,  and 
will  describe  reuse  techniques  that  have  been  accomplished 
to  date. 
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Chapter  III  discusses  the  NASA  System  Reference  Model 
(SRM)  and  the  use  of  assertions  in  repository  libraries. 

Chapter  IV  discusses  the  use  of  patterns  to  facilitate 
Software  reuse. 

Chapter  V  provides  conclusions  and  recommendations  for 


future  research. 


II.  IV&V  AND  SOFTWARE  REUSE 


A.  COMPLEXITY  OF  SOFTWARE  DESIGN 

Anyone  who  is  associated  with  software  design 
understands  that  software  systems  can  be  extremely  complex. 
They  are  so  complex  that  most  software  engineering 
researchers  often  focus  their  research  solely  on  ways  to 
deal  with  complexity.  The  reason  these  systems  are  so 
complicated  can  often  be  traced  back  to  the  users' 
requirements  for  that  system.  A  system  that  is  required  to 
perform  several  functions  will  naturally  be  more  complex 
than  a  system  that  is  required  to  perform  just  one  function. 
Common  problems  that  occur  when  developing  software  include 
failing  to  match  the  final  product  to  the  customers'  needs, 
or  dealing  with  errors  in  the  software  that  often  reveal 
themselves  at  the  worst  possible  time  and  are  often  costly 
to  fix.  One  way  to  try  to  avoid  these  problems  is  to 
implement  Independent  Verification  and  Validation  (IV&V)  and 
software  reuse  into  the  development  of  the  systems.  This 
chapter  covers  IV&V,  how  IV&V  is  conducted,  and  how  software 
engineers  are  currently  leveraging  software  reuse  in 

building  software  systems. 

B.  DEFINITIONS 

Independent :  Independence  in  relation  to  IV&V  is 

defined  by  the  Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  using  three  parameters:  technical 

independence,  managerial  independence,  and  financial 

independence . 
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Technical  Independence:  "requires  the  V&V  effort  to 
utilize  personnel  who  are  not  involved  in  the  development  of 
the  software."® 

Managerial _ Independence :  "requires  that  the 

responsibility  for  the  IV&V  effort  be  vested  in  an 
organization  separate  from  the  development  and  program 
management  organizations."^ 

Financial  Independence:  "requires  that  control  of  the 
IV&V  budget  be  vested  in  an  organization  independent  of  the 
development  organization . "i® 

Verification:  "The  process  of  evaluating  a  system  or 
component  to  determine  whether  the  products  of  a  given 
deployment  phase  satisfy  the  conditions  imposed  at  the  start 
of  that  phase."  Software  verification  answers  the 
question,  "Are  we  building  the  product  right?" 

Validation:  "The  process  of  evaluating  a  system  or 
component  during  or  at  the  end  of  the  development  process  to 
determine  whether  it  satisfies  specified  requirements." 
^^Software  validation  answers  the  question  "Are  we  building 
the  right  product?" 


°  Institute  of  Electrical  and  Electronics  Engineers,  Standard  for 
Software  Verification  and  Validation, IEEE-STD-1012,  June  08,  2005. 

9  Ibid. 

Ibid. 

Ibid. 

Ibid. 
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Software  IV&V:  "a  series  of  technical  and  management 
activities  performed  by  someone  other  than  the  developer  of 
a  system  to  improve  the  quality  and  reliability  of  that 
system  and  to  assure  that  the  delivered  product  satisfies 
the  user's  operational  needs. 

Software  Reuse:  "the  use  of  existing  software  artifacts 
in  the  development  of  other  software  artifacts  with  the  goal 
of  improving  productivity  and  quality,  among  other 
factors  . 

Requirement _ Specification  :  "an  organization's 

understanding  (in  writing)  of  a  customer  or  potential 
client's  system  requirements  and  dependencies  at  a 
particular  point  in  time  (usually)  prior  to  any  actual 
design  or  development  work."^^ 

Pattern :  "a  body  of  literature  to  help  software 
developers  resolve  recurring  problems  encountered  throughout 
all  of  software  development."^® 

C .  IV&V  BACKGROUND 

In  the  early  1940s  the  first  computer  was  developed  to 
calculate  artillery  firing  tables  for  the  United  States 

R.  Lewis,  Independent  verification  &  validation:  A  life  cycle 
engineering  process  for  quality  software .  New  York:  John  Wiley  &  Sons, 
(1992) :  xxiii . 

W.  Lim,  Managing  software  reuse,  a  comprehensive  guide  to 
strategically  reengineering  the  organization  for  reusable  components . 
Upper  Saddle  River,  New  Jersey:  Prentice  Hall  PTR,  (1998)  :  7  . 

D.  Le  Vie,  "Writing  software  requirements  specifications"  TECHWR-L 
(MAR  2007)  http://www.techwrl.com/techwhirl/magazine/writing/ 
softwarerequirementspecs.htm  (accessed  July  15,  2008). 

B.  Appleton,  "Patterns  and  software:  essential  concepts  and 
terminology"  CM  Crossroads  (2000),  http://www.cmcrossroads.com/ 
bradapp/docs/patterns-intro . html  (accessed  September  20,  2008). 
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Army's  Ballistic  Research  Laboratory.  The  design  of  the 
computer  was  focused  primarily  on  hardware,  not  paying  much 
attention  to  software.  In  fact,  some  would  say  in  the  early 
stages  of  computing,  software  was  often  ignored.  The 
intention  of  computers  at  this  time  was  to  perform  a  single 
task.  When  the  task  was  identified  the  computers  were  hard¬ 
wired  to  accomplish  that  task.  With  the  role  of  software 
being  so  small  the  need  for  IV&V  had  not  yet  been 
recognized.  However,  as  time  passed  and  the  role  and  cost  of 
software  grew,  the  need  for  IV&V  became  evident. 

In  the  mid  1940s  John  Von  Neumann  came  up  with  two 
concepts  that  would  have  a  direct  impact  on  software  design. 
The  first  was  known  as  "shared  program  technique."  "This 
technique  states  that  the  actual  computer  hardware  should  be 
simple  and  not  need  to  be  hand-wired  for  each  program. 
Rather,  complex  instructions  should  be  used  to  control  the 
simple  hardware,  allowing  it  to  be  reprogrammed  much 
faster . 

The  second  concept  he  developed  was  called  "conditional 
control  transfer."  "This  idea  gave  rise  to  the  notion  of 
subroutines,  or  small  blocks  of  code  that  could  be  jumped  to 
in  any  order,  instead  of  a  single  set  of  chronologically 
ordered  steps  for  the  computer  to  read.  The  second  part  of 
the  idea  stated  that  computer  code  should  be  able  to  branch 
out  based  on  logical  statements  such  as  "IF"  (expression) 
"THEN, "  and  looped  with  others  such  as  a  "FOR" 


17  C.  Robat,  "Introduction  to  software  history."  The  History  of 
Computing  Project  (October  17,  2006),  http://www.thocp.net 
/ software/software_reference/ introduction_to_s of tware_hi story . htm 
(accessed  June  11,  2008) . 
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statement."^®  The  use  of  these  concepts,  and  others  like 
them,  allowed  software  to  grow  into  a  more  significant  part 
of  computer  design. 

As  software  grew,  so  did  the  cost  associated  with  it. 
In  the  1950s,  software's  cost  was  only  20%  of  the  overall 
system  cost.  In  the  1980' s,  software  costs  rose  to  80%. 
Today,  software  costs  can  be  up  to  95%  of  the  overall  system 
cost.^®  These  rising  costs  forced  software  developers  to 
find  a  way  to  save  money. 

In  the  late  1950s,  one  of  the  leading  software 
developers  was  the  Department  of  Defense  (DoD)  .  The  DoD 
began  to  notice  projects  were  consistently  behind  schedule, 
over  budget,  and  did  not  provide  the  required  performance. 
This  was  unacceptable  not  only  for  financial  reasons  but 
because  software  errors  can  lead  to  loss  of  life,  injury,  or 
loss  of  property  especially  in  military  systems.  The  DoD  was 
repeatedly  surprised  by  the  costly  projects  because 
"...software  development  contractors  often  gave  overly 
optimistic  assessments  of  the  software  development  status  to 
the  DoD.  "2®  To  address  this,  the  DoD  launched  a  plan  to 
conduct  IV&V  on  their  software  systems  in  an  attempt  to  get 
accurate  assessments  of  how  their  projects  were  doing.  The 


C.  Robat,  "Introduction  to  software  history."  The  History  of 
Computing  Project  (October  17,  2006),  http://www.thocp.net 
/ software/ software_reference/ introduction_to_software_hi story . htm 
(accessed  June  11,  2008) . 

S.  Reiss,  A  practical  introduction  to  software  design  with  C++. 
New  York:  John  Wiley  &  Sons,  1998,  397-421. 

20  s.  Rakin,  "Food  for  thought:  What  is  software  quality  assurance?" 
Software  Quality  Consulting  (Jan.  2005,  Vol.2  No.l), 

http : / / WWW . swqual . com/ newsletter /vol2 / nol / vol2nol . html  (accessed  June 
01,  2008) . 
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first  program  to  use  IV&V  was  the  Atlas  Missile  Program  in 
the  late  1950s.  An  independent  software  tester  was  hired  to 
conduct  unbiased  testing  of  the  software. 

Over  time,  the  role  of  IV&V  continued  to  develop  and  in 
the  1970' s  "...  the  U.S.  Army  sponsored  the  first 
significant  such  IV&V  program  for  the  Safeguard  Anti- 
Ballistic  Missile  System. "22  The  program  was  designed  to 
identify  and  eliminate  the  high  risks  that  are  common  with 
military  systems.  It  was  successful  in  meeting  its  goal  and 
"By  the  mid-  to  late  1970' s,  IV&V  was  rapidly  becoming 
popular  and  in  some  cases  was  required  by  the  military 
services ...  "23  "it  was  from  this  effort  that  IV&V  became 
well  known  within  the  Department  of  Defense  and  the 
aerospace  communities  as  an  accepted  method  of  ensuring 
better  quality,  performance,  and  reliability  of  critical 
systems  .  "24 

In  the  decades  following  the  seventies,  IV&V  became  an 
intricate  part  of  the  software  development  process.  A 
process  that  started  as  "...mostly  free-form,  not  very 
independent,  often  started  too  late  to  be  really  effective, 
and  was  sometimes  even  performed  by  the  very  people  who  were 
developing  the  system... "23  grew  into  process  where  "...a 

9  1 

S.  Rakin,  "Food  for  thought:  What  is  software  quality  assurance?" 
Software  Quality  Consulting  (Jan.  2005,  Vol.2  No.l), 

http : // WWW . swqual . com/ newsletter/vol2/ nol/vol2nol . html  (accessed  June 
01,  2008) . 

22  R.  Lewis,  Independent  verification  &  validation:  A  life  cycle 
engineering  process  for  quality  software .  New  York:  John  Wiley  &  Sons, 
1992,  xxiii. 

22  Ibid. 

24  Ibid. 

22  R.  Lewis,  Independent  verification  &  validation:  A  life  cycle 
engineering  process  for  quality  software .  New  York:  John  Wiley  &  Sons, 
(1992) :  xxiii . 
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completely  independent  entity  evaluates  the  work  products 
generated  by  the  team  that  is  designing  and/or  executing  a 
given  pro ject . . . The  independent  entity  will  also 
"...monitor  and  evaluate  every  aspect  of  the  project  itself 
from  inception  to  completion . 

While  the  cost  of  conducting  IV&V  is  high,  the  money 
saved  by  preventing  errors  and  rework  is  far  greater.  In 
1993,  the  National  Aeronautics  and  Space  Administration 
(NASA)  established  an  IV&V  facility  in  the  wake  of  the  Space 
Shuttle  Challenger  accident.  The  facility  was  developed  as 
part  of  a  plan  "to  provide  the  highest  achievable  levels  of 
safety  and  cost-effectiveness  for  mission  critical 
software. "In  2006,  NASA  allocated  $27  Million  to  the 
IV&V  Facility  Budget,  of  which  $19  Million  went  directly  to 
IV&V  Services. After  conducting  a  Return  on  Investment 
analysis,  "NASA  realized  a  software  rework  risk  reduction 
benefit  of  $1.6  Billion  in  Fiscal  Year  2006  alone. From 
the  facilities  inception  at  NASA,  it  has  experienced 
continued  growth  while  providing  better  software/system 
performance,  higher  confidence  in  the  software  reliability, 
and  a  reduced  maintenance  cost. 


C.  Nickolett,  "Project  due  diligence:  independent  verification  and 
validation."  White  Paper . Comprehensive  Consulting  Solutions.  Mar  2001: 
1-6.  http://www.comp-soln.com/lVV_whitepaper.pdf  (accessed  June  01, 

2008)  . 

Ibid. 

National  Aeronautics  and  Space  Administration  (NASA) ,  "NASA  iV&V 
facility  -  about  IV&V."  National  Aeronautics  and  Space  Administration 
(NASA),  http://www.nasa.gov/centers/ivv/  about/index . html  (accessed  June 
01,  2008) . 

29  NASA  IV&V  Facility,  "NASA  IV&V  2006  annual  report."  NASA  IV&V 
Facility,  http:// www . nasa . gov/ centers/ ivv/ pdf/ 

174321main_Annual_Report_06_Final.pdf  (accessed  June  01,  2008). 

29  Ibid. 
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When  performed  correctly  IV&V  can  be  a  crucial  part  of 
the  software  development  process.  The  process  begins  with 
developing  Software  Integrity  Levels  (SILs)  which  "are  a 
range  of  values  that  represent  software  complexity, 
criticality,  risk,  safety  level,  security  level,  desired 
performance,  reliability,  or  other  project-unique 
characteristics  that  define  the  importance  of  the  software 
to  the  user  and  acquirer. "3i  siLs  are  then  used  to  determine 
which  V&V  tasks  to  perform.  The  higher  the  software 
integrity  level,  the  more  V&V  tasks  assigned.  SILs  are  not 
constant  and  can  change  as  software  evolves  to  ensure  the 
appropriate  V&V  tasks  are  being  performed.  Below  is  an 
example  of  SILs  based  upon  the  concepts  of  consequences  and 
mitigation  potential  as  well  as  an  example  of  V&V  processes, 
activities,  and  tasks  from  the  IEEE  Standard  for 
Verification  and  Validation.  These  examples  are  provided  as 
guidance  on  how  software  developers  can  incorporate  IV&V 
into  their  software  design  to  assist  in  reducing 
specification  errors. 


Institute  of  Electrical  and  Electronic  Engineers,  Standard  for 
Software  Verification  and  Validation, IEEE-STD-1012,  June  08,  2005. 
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Description  of  Software  integrity  Level  Level 

Software  element  must  execute  correctly  or  grave  consequences  (loss  of  life,  loss  of 

system,  economic  or  social  loss)  will  occur.  No  mitigation  is  possible.  4 

Software  element  must  execute  correctly  or  the  intended  use  (mission)  of  the  system/ 
software  will  not  be  realized,  causing  serious  consequences  (permanent  injury,  major 
system  degradation,  economic  or  social  impact).  Partial  to  complete  mitigation  is  possible.  3 

Software  element  must  execute  correctly  or  an  intended  function  will  not  be  realized, 

causing  minor  consequences.  Complete  mitigation  possible.  2 

Software  element  must  execute  correctly  or  intended  function  will  not  be  realized, 

causing  negligible  consequences.  Mitigation  not  required.  1 

Figure  1.  Examples  of  Software  Integrity  Levels 
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Figure  2.  V&V  processes,  activities,  and  tasks. 
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D. 


CURRENT  GUIDANCE  FOR  VALIDATION 


Incorporating  IV&V  into  software  design  is  essential  to 
reducing  specification  errors.  What  software  engineers  need 
to  ensure  is  when  IV&V  is  applied  it  is  done  so  correctly. 
The  definitions  for  V&V  provided  at  the  beginning  of  this 
chapter  allow  for  the  use  of  computer-based  V&V  tools  to 
check  the  correctness  of  a  system  or  a  specific  component 
against  a  formal  specification  derived  from  the  natural 
language  requirements.  The  specifications  are  created  and 
the  final  product  is  then  built  to  satisfy  those 
specifications.  Validation  that  is  being  conducted  in 
accordance  with  the  guidelines  provided  by  the  IEEE 
evaluates  specific  components  or  the  final  product  with  the 
specifications.  This  process  is  in  fact  verification.  The 
product  is  being  built  correctly  according  to  the 
specifications,  however,  it  is  not  known  if  the 
specifications  themselves  are  correct.  It  is  imperative  that 
validation  be  conducted  on  the  specifications  that  are 
created  to  ensure  that  the  requirements  for  the  project  are 
understood  and  that  the  correct  product  is  built. 

E .  SOFTWARE  REUSE 

Software  reuse  is  a  practice  that  began  in  the  1950s 

with  the  goal  of  improving  software  development  productivity 

and  quality.  For  the  past  twenty  years  a  great  deal  of 

research  has  been  focused  on  software  reuse  and  its  role  in 

software  design.  Areas  that  have  been  given  attention 

include  but  are  not  limited  to  reuse  libraries,  design 

patterns,  and  reuse  using  formal  specifications  of 

requirements.  While  software  reuse  holds  promise  of 
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improving  software  development  productivity  and  software 
quality,  the  success  of  reuse  is  based  on  the  quality  of  the 
reusable  artifacts.  The  reuse  of  software  that  has  not  been 
verified  and  validated  contradicts  the  intended  goal  of 
producing  quality  software  because  errors  in  the  software 
may  still  exist.  This  reasoning  also  holds  true  when 
discussing  the  use  of  formal  requirements  specifications. 
The  use  of  formal  requirements  specifications  is  essential 
in  the  automation  of  the  software  verification  process. 
However,  we  assert  that  the  correctness  of  these  formal 
specifications  must  be  first  validated  before  they  can  be 
used  to  verify  correctness  of  the  software. 

Formal  specification  has  been  an  active  area  of 
research  for  more  than  two  decades.  The  requirements 
specification  of  a  software  component  describes  the  expected 
functions  and  behavior  of  the  software.  The  ability  to  reuse 
the  software  component  becomes  evident  if  its  structure  and 
behavior  are  compatible  with  new  software  being  designed. 

Verification  has  been  another  popular  research  topic 
for  over  20  years.  Automated  finite  state  verification  tools 
have  been  developed  to  assist  software  developers  in 
verifying  system  specifications.  The  users  of  these  tools 
must  be  capable  of  specifying  the  requirements  of  the  system 
they  are  developing  in  the  specification  language  the  tool 
understands.  Behavior  for  a  software  component  is  typically 
specified  using  temporal  logic  in  an  attempt  to  avoid  the 
ambiguity  derived  from  natural  language. 
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F. 


FORMAL  SPECIFICATION  PATTERNS 


To  assist  developers  in  specifying  the  behavior  in  a 
temporal  logic,  Dwyer  suggests  the  use  of  property 
specification  patterns.  "A  property  specification  pattern  is 
a  generalized  description  of  a  commonly  occurring 
requirement  on  the  permissible  state/event  sequences  in  a 
finite  state  model  of  a  system. "^2  These  patterns  describe 
the  essential  behavior  of  a  system  and  provide  expressions 
of  this  behavior  in  a  range  of  common  temporal  logics  to  be 
used  with  verification  tools.  The  patterns  are  then  given 
distinct  names  describing  their  behavior  which  allows  them 
to  be  mapped  to  examples  of  known  use,  to  relationships  to 
other  patterns,  and  to  specific  formalisms.  To  facilitate 
verification,  Dwyer  proposes  the  development  of  a  system  of 
property  specific  patterns  for  finite  state  verification 
tools.  The  system  is  a  set  of  patterns  or  library  organized 
into  one  or  more  hierarchies,  with  connections  between 
related  patterns  to  facilitate  the  browsing  of  the  system. 
"A  user  would  search  for  the  appropriate  pattern  to  match 
the  requirement  being  specified,  use  the  mapping  section  to 
obtain  the  essential  structure  of  the  pattern  in  the 
formalism  used  by  a  particular  (verification)  tool,  and  then 
instantiate  that  pattern  by  plugging  in  the  state  formula  or 
events  specific  to  the  requirement .  "^3  The  use  of  these 
patterns  allows  for  the  specification  of  critical  properties 


32  M.  Dwyer,  G.  Avrunin,  et.al.  "Patterns  in  property  specifications 
for  finite-state  verification."  Proceedings  of  the  21st  international 
conference  on  software  engineering  (1999)  :  411-420. 

33  Ibid. 
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that  exist  in  software  systems  and  guides  users  of 
verification  tools  to  express  these  properties  in  a 
specification  language. 

In  2005,  Konrad  and  Cheng  went  a  step  further  with 
specification  patterns  and  introduced  real-time 
specification  patterns  that  can  be  used  to  specify  real-time 
properties  for  embedded  systems.  Similar  to  Dwyer's 
specification  patterns,  the  real-time  specification  patterns 
contain  templates  for  specifying  real-time  properties  in 
terms  of  real-time  temporal  logic. ^4  This  pattern  system  is 
intended  to  provide  strategies  for  specifying  real-time 
properties  in  a  formal  specification  language,  where  the 
properties  are  amenable  to  automated  analysis  such  as 
verification  tools. ^5 

Specification  patterns  and  the  use  of  libraries  to 
store  those  patterns  provide  another  form  of  software  reuse. 
This  form  of  reuse  aims  at  reducing  the  cost  and  improving 
the  quality  of  formal  specification  development.  However, 
the  effectiveness  of  the  specification  pattern  reuse  depends 
on  the  correctness  and  consistency  of  the  resultant 
requirements.  Proper  validation  needs  to  be  performed  in 
order  to  confirm  that  the  requirements  are  understood. 

Otani  et  al .  explains  a  concept  of  developing  and 
validating  libraries  of  temporal  formal  specifications. 
These  libraries  would  include  UML  Statechart  based 
assertions  for  formal  specifications  and  their  associated 


B.  Cheng  and  S.  Konrad,  "Real-time  specification  patterns." 
Proceedings  of  the  27th  international  conference  on  software  engineering 
(2005) :  372-381 . 

Ibid. 
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validation  test  scenarios.^®  We  intend  to  build  the 
validation  test  scenarios  with  the  goal  of  ensuring  that 
specifications  within  the  libraries  are  indeed  error-free 
and  consistent.  The  following  chapter  describes  the  NASA 
System  Reference  Model  (SRM)  and  its  role  in  capturing  a 
modeler's  understanding  of  a  specific  problem. 


T.  Otani,  D.  Drusinsky,  et.al.  "Validating  UML  statechart  based 
assertions  libraries  for  improved  reliability  and  assurance." 
Proceedings  of  the  Second  International  Conference  on  Secure  System 
Integration  and  Reliability  Improvement  (SSIRI  2008),  Yokohama,  Japan, 
July  14-17,  2008,  47-51. 
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III.  SYSTEM  REFERENCE  MODEL 


A.  BACKGROUND 

The  National  Aeronautics  and  Space  Administration 
(NASA)  has  continuously  developed  their  IV&V  program, 
supporting  new  technologies  and  better  validation  and 
verification  techniques  in  an  effort  to  improve  the 
validation  and  verification  process.  Earlier  versions  of  the 
V&V  process  included  the  Criticality  and  Risk  Assessment 
(CARA)  and  the  Software  Integrity  Level  Assessment  Process 
(SILAP) .  Both  processes  were  found  lacking  because  they 
relied  on  manual  examination  and  independent  testing  of 
target  code.  These  techniques  are  ineffective  for  use  in 
validation  because  there  are  no  links  from  the  requirements 
to  the  system's  features,  capabilities,  properties  and 
functions.  Without  formal  specifications  of  the  system 
behaviors  both  CARA  and  SILAP  were  unable  to  validate  the 
correctness  and  completeness  of  the  developer' s 
understanding  of  the  requirements.  Finally,  the  processes 
were  unable  to  locate  the  subtle  errors  in  increasingly 
complex  software-intensive  system.  Both  CARA  and  SILAP 
evaluated  the  risk  of  software  components  in  a  system  by 
compiling  a  list  of  software  components  and  evaluating  them 
to  prioritize  risk  assessment,  which  cannot  show  that  the 
system  being  built  is  the  correct  system.  NASA  is  in  the 
process  of  replacing  SILAP  with  advanced  computer-aided 
validation  techniques. 
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The  NASA  IV&V  Facility  recognized  a  need  for  validation 
to  be  more  than  a  risk  assessment;  it  needed  to  provide  a 
model  for  the  system  to  show^'^ : 

•  What  the  system  is  supposed  to  do. 

•  What  the  system  is  not  supposed  to  do  and 

•  How  the  system  should  respond  under  adverse  conditions. 

The  NASA  IV&V  Facility  now  relies  on  the  use  of  a 
System  Reference  Model  (SRM)  for  each  product.  "The  SRM 
provides  the  basis  for  validating  the  completeness  and 
correctness  of  the  targeted  requirements  set. Once  the 
targeted  requirements  are  developed  the  independent 
validation  team  is  able  to  validate  those  requirements.  The 
SRM  supports  a  computer-aided  validation  technique  through 
which  the  independent  validation  team' s  understanding  and 
perception  of  the  problem  is  validated  through  the  team' s 
representation  of  the  SRM' s  features,  properties,  function, 
and  capabilities.  It  is  also  during  this  time  that  the 

development  team  is  able  to  discover  and  correct  any 
identified  problems  or  concerns  with  their  understanding  of 
the  requirements  for  the  intended  system.  This  is  important 
because  the  model  holds  the  responsibility  to  be  complete 
and  accurate  to  serve  its  intended  purpose  and  the 

development  team  holds  the  responsibility  to  ensure  that  the 
model  fulfills  that  purpose. 


3”^  K.  Woodham,  System  Reference  Model  (SRM)  development  and  analysis 
guideline,  1st  draft  (National  Aeronautics  and  Space  Administration 
(NASA),  2007). 

Ibid. 
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B.  DEFINITIONS 

The  following  definitions  are  described  in  the  SRM 
guideline39.  The  definition  of  dependability  will  be 
customized  to  the  user's  needs  and  wants  of  the  system. 

•  Dependability :  A  dependable  system  is  one  that  provides 
the  appropriate  levels  of  correctness  and  robustness  in 
accomplishing  its  mission  while  demonstrating  the 
appropriate  levels  of  availability,  consistency, 
reliability,  safety,  and  recoverability. 

•  Availability :  The  probability  that  a  system  is 

operating  correctly  and  is  ready  to  perform  its  desired 
functions . 

•  Consistency :  The  property  that  invariants  will  always 
hold  true  in  the  system. 

•  Correctness :  A  characteristic  of  a  system  that 

precisely  exhibits  predictable  behavior  at  all  times  as 
defined  by  the  system  specifications. 

•  Reliability :  The  property  that  a  system  can  operate 
continuously  without  experiencing  a  failure. 

•  Robustness :  A  characteristic  of  a  system  that  is 
failure  and  fault  tolerant. 

•  Safety :  The  property  of  avoiding  a  catastrophic  outcome 
given  a  system  fails  to  operate  correctly. 

•  Recoverability :  The  ease  for  which  a  failed  system  can 
be  restored  to  operational  use. 

C.  SYSTEM  REFERENCE  MODEL  DEVELOPMENT 

Without  a  doubt,  any  process  can  become  overwhelming  in 
both  cost  and  time.  Thus,  it  is  necessary  for  the  SRM  to 
have  an  appropriate  level  of  specificity  so  that  a 
completion  point  can  be  reached.  "The  appropriate  level  of 


39  K.  Woodham,  System  Reference  Model  (SRM)  development  and  analysis 
guideline,  1st  draft  (National  Aeronautics  and  Space  Administration 
(NASA),  2007). 
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V&V  is  a  function  of  the  time  available  to  do  the  V&V 


evaluations,  and  this  should  in  turn  be  a  function  of  the 
risk  that  will  be  incurred  if  the  V&V  is  not  done,  or  the 
risk  that  will  be  mitigated  if  a  given  level  of  V&V  is 
done."^°  The  SRM  still  must  be  developed  to  a  level  of 
fidelity  to  support  validation  of  the  system  and  result  in 
completeness  and  correctness  of  the  targeted  requirements. 

The  SRM  can  be  extremely  detailed  and  can  consist  of 
high-level  use  cases.  Unified  Modeling  Language  (UML) 
artifacts  such  as  activity  diagrams,  sequence  diagrams  and 
object  class  diagrams,  and  a  set  of  formal  assertions  to 
describe  precisely  the  necessary  behaviors  to  satisfy  system 
goals,  with  respect  to  the  three  questions  stated 
previously.  These  many  artifacts  allow  the  team  to  properly 
express  the  requirements  through  the  SRM  and  ensure  that 
their  understanding  of  the  requirements  is  correct. 

The  development  of  the  SRM  begins  with  a  scoping 
period.  During  this  time  the  SRM  development  team  commences 
with  a  front-end  analysis.  The  front-end  analysis  ensures 
that  the  team  has  a  clear  perspective  of  the  intended  use  of 
the  model.  This  high-level  abstraction  helps  the  team  ensure 
that  the  model  is  defined  which  in-turn  drives  the 
objectives  of  the  model  development.  The  scoping  period  also 
ensures  that  the  SRM  development  is  based  on  concept-level 
documentation  rather  than  requirements  generated  by  the 


R.  Logan  and  C.  Nitta,  "Verification  &  validation:  process  and 
levels  leading  to  qualitative  or  quantitative  validation  statements." 
SAE  Transactions  vol.113,  no. 5  (2004),  http://bill.cacr.caltech.edu 
/valworkshop/upload/f iles/UCRLTR-200131sae04 f a . pdf  (accessed  June  01, 
2008)  . 
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system  developers.  Finally,  the  scoping  period  should 
finalize  the  level  of  specificity  of  the  requirements  so 
that  a  completion  point  can  be  reached. 

The  scoping  period  consists  of  analyzing  the 
constraints,  restrictions  and  targeted  tasks  and 
requirements  to  recognize  the  depth  of  the  modeling  needed. 
Additionally,  requirements  that  will  not  be  modeled  in  the 
SRM  are  identified  and  the  team  ensures  that  sufficient 
concept  documentation  is  available  to  continue.  The  concept 
documents  used  during  the  process  are  found  in  many  forms  of 
stakeholder  inputs  from  mission  statements  to  concepts  of 
operations.  The  scoping  period  ends  with  a  clear 
understanding  of  the  system  elements  that  need  to  be 
addressed  and  the  depth  that  they  need  to  be  defined.  The 
level  of  fidelity  should  be  determined  at  this  point  to 
ensure  completeness  and  correctness  of  the  targeted  system 
requirements . 

The  next  stages  of  the  SRM  development  are  accomplished 
through  the  development  of  use  cases  and  UML  artifacts  as 
well  as  supporting  assertions.  The  SRM  team,  using  the 
conceptual  documents,  will  begin  by  documenting  system 
behaviors.  It  is  during  this  time  that  the  system  goals 
should  be  identified  and  a  traceability  matrix  developed, 
populated  with  these  top-level  goals.  Additionally,  the 
operational  environment  must  be  identified  and  the 
traceability  matrix  should  be  populated  with  operation 
environment  characteristics  that  need  to  be  addressed  by  the 
system  model.  The  top-level  use  cases  developed  to  address 
the  overall  system  goals  are  peer  reviewed  to  validate  that 
the  preliminary  use  case  set  spans  the  high-level 
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description  of  the  system  and  is  documented  in  the 
traceability  matrix.  The  top-level  use  cases  are  abstracted 
from  the  details  of  the  system  and  are  goal-oriented.  These 
use  cases  help  the  developers  to  get  a  clear  understanding 
of  the  process  and  problems  to  be  solved  as  well  as  the 
goals  and  objectives  of  the  system.  The  top-level  use  cases 
then  are  refined  into  lower-level  use  cases  and  activity 
diagrams  which  can  be  mapped  to  sequence  diagrams.  The 
process  continues  to  become  more  specific  to  ensure  that  the 
goals  and  objectives  are  accomplished  but  also  to  verify 
that  their  constraints  are  also  adequately  captured.  The 
diagrams  should  provide  a  complete  representation  of  the 
behavior  expected  to  be  displayed  by  the  system. 
Additionally,  all  behaviors  should  be  mapped  and  defined 
into  the  traceability  matrix  and  peered  reviewed  to  ensure 
correctness.  The  overall  goal  is  to  ensure  that  the  top- 
level  use  cases  have  been  refined  into  detailed  lower-level 
uses  cases  that  represent  not  only  the  Main  Success  Scenario 
(MSS)  but  are  fully  elaborated  to  ensure  necessary 
extensions  are  also  represented.  Finally,  the  modeling  team 
has  to  ensure  that  any  dependability  considerations  are 
addressed  and  represented  in  the  model.  This  entire  effort 
should  represent  the  desired  system  behaviors  as  well  as  any 
necessary  extensions  and  assertions  that  map  to  the  top- 
level  goals  and  requirements.  The  model  is  ready  to  be 
validated . 

D.  VALIDATING  THE  SYSTEM  REFERENCE  MODEL 

The  newly  developed  SRM  is  a  representation  created  by 
the  SRM  development  team  and  is  a  result  of  the  teams  own 
perceptions  and  understanding  of  the  desired  system 
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behaviors.  As  such  the  representation  could  be  wrong  if  the 
team  misinterpreted  the  desired  behaviors  of  the  system. 
This  is  why  the  SRM  must  be  validated  to  reduce 
specification  error  as  well  as  to  ensure  that  the  behavior 
requirements  created  by  the  SRM  development  team  are 
measured  against  the  SRM  for  correctness.  The  SRM  is  a  model 
of  the  intended  system  and  it  must  meet  any  dependable 
considerations  in  order  for  the  intended  system  to  be  so  as 
well . 

The  validation  process  is  twofold  and  can  begin  with  a 
formal  review  and  tracing  of  the  UML  artifacts  to  include: 
use  case  definitions  and  models,  supporting  assertions,  and 
activity  diagrams.  Other  artifacts  reviewed  include  the 
complete  set  of  system-behavior  definitions  based  on 
stakeholder  goals  and  system  constraints  and  operations 
environments  defined  in  the  concept  documentation.  During 
this  review  the  formal  tracing  of  the  requirements  from  the 
top-level  to  the  more  refined  lower-levels  and  the  activity 
diagrams  and  sequence  diagrams  helps  to  identify  the 
subsystems  and  components  responsible  for  the  system 
requirements.  Additionally,  during  this  process  all  the 
requirements  are  elicited  and  peer  reviewed.  This  ensures 
that  all  targeted  requirements  have  been  identified  and 
traced  through  the  artifacts.  During  this  time  all  necessary 
objects  and  events  are  labeled,  identified,  and  checked  to 
ensure  there  are  no  unnecessary  objects  or  events.  The  above 
process  ensures  that  all  targeted  requirements  are  fully 
detailed  in  accordance  with  their  goals.  During  each  review 
each  step  is  subjected  to  extensive  group  review  to  validate 
that  the  SRM  is  a  complete  and  unambiguous  representation  of 
the  system. 
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The  second  step  of  the  validation  process  is  to  execute 
as  much  of  the  model  as  possible  through  computer-aided 
auditing.  Run-time  verification  of  formal  assertions  is  able 
to  check  for  inconsistency,  omission  and  errors  in  the  SRM. 
By  executing  as  much  of  the  model  as  possible  it  increases 
the  evidence  that  the  model  being  developed  is  the  correct 
system.  The  independent  validation  team  is  able  to  use  the 
evidence  of  validation  to  ensure  that  the  SRM  is  the  correct 
system . 

The  IV&V  team' s  requirements  elicitation  and  validation 
tasks  produce  deliverable  packages,  consisting  of:  UML 
models  for  reference  model  constituents,  natural  language 
assertions,  formal  representation  of  the  assertions,  and  a 
validation  test  suite  for  each  assertion.  The  test  suites 
are  detailed  and  include  tests  that  cover  multiple  scenarios 
that  meet  the  requirements  of  the  assertions,  and  will  be 
discussed  further  in  the  next  chapter.  These  deliverable 
packages  are  the  evidence  gathered  to  decrease  specification 
errors  and  must  be  done  to  validate  the  SRM  and  provide 
evidence  of  dependability  of  the  system. 

The  SRM  is  intricate  and  detailed  in  order  to  show  its 
dependability.  But  before  dependability  can  be  shown  the  SRM 
assertions  must  be  validated  to  decrease  specification 
errors.  In  fact  the  assertions  should  precisely  model  the 
required  behavior  of  the  system  and  if  they  are  able  to  do 
so  the  model  is  on  its  way  to  being  validated.  But 
assertions  also  have  to  be  validated  and  are  validated 
through  an  execution-based  model  checker  for  dependability 
of  the  model  under  test. 
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E.  INCREASING  THE  USABILITY  OF  THE  SRM 

The  difficulty  with  assertions  is  in  their  creation.  It 
not  only  takes  time  and  effort,  but  the  correctness  of  the 
executable  assertions  depends  on  the  ability  of  the  modelers 
to  specify  correct  assertions.  It  is  difficult  to  specify 
and  develop  correct  assertions.  The  modelers  must  have  a 
correct  representation  of  the  structure  and  behavior  of  the 
SRM,  the  assertions  must  also  be  correct.  If  faulty 
assertions  are  used  they  are  not  effective  in  the  IV&V 
process.  We  believe  that  a  library  built  with  correct 
assertions  would  enable  the  assertions  to  be  reused.  This 
could  both  decrease  the  burden  on  the  modelers  to  develop 
the  assertions  and  improve  the  ability  of  the  independent 
validation  teams  to  validate  the  dependability  of  the 
software . 
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IV.  BUILDING  AN  ASSERTION  LIBRARY 


A.  BACKGROUND 

As  mentioned  in  the  previous  chapters  the  SRM  is  a 
representation  created  by  the  developers  as  a  result  of 
their  own  understanding  of  the  desired  system  behaviors.  The 
SRM  must  be  validated  to  reduce  specification  errors.  One  of 
the  ways  to  do  this  is  through  assertions  which  precisely 
model  the  required  behavior  of  the  system  and  are  the 
foundation  of  the  SRM.  Through  testing  and  modeling 
assertions  the  independent  validation  team  begins  to 
comprehend  the  problem  domain  and  refine  any  problems  to 
ensure  that  the  SRM  meets  the  user' s  requirements  and  the 
correct  system  is  built.  The  current  way  to  build  assertions 
is  to  develop  the  assertions  from  natural  a  language 
description  of  the  user's  understanding  anew  every  time; 
this  can  be  a  time-consuming  and  error-prone  undertaking.  We 
believe  that  an  assertion  library  can  help  ease  these  tasks 
by  providing  validated  assertions  which  can  be  reused. 

The  purpose  of  this  chapter  was  to  construct  an 
approach  to  building  an  assertion  library  with  a  small 
number  of  assertions  that  have  been  validated  for 
correctness  and  are  reusable.  We  define  a  library  to  be  a 
collection  of  assertions  that  are  stored,  collectively 
shared  and  can  be  filled  with  more  assertions  as  needed.  The 
assertions  in  the  library  are  validated  through  the  use  of 
test  scenarios  that  we  designed.  The  test  scenarios  are 
patterns  which  test  the  assertion  for  the  required  behavior. 
The  purpose  of  the  test  suites  is  to  disambiguate  the 
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assertions,  and  test  for  correctness  meaning  that  the 
assertions  accurately  reflect  the  natural  language  statement 
as  we  intended. 

The  assertion  library  would  be  built  so  that  the 
assertions  are  reusable  and  adaptable  for  future  projects. 
Software  developers  can  select  from  the  library  any 
assertions  that  meet  their  needs  and  adapt  them  or  use  them 
as  an  example  to  build  their  own.  In  each  case  we  ensured 
that  the  assertion  was  general  to  increase  the  ability  to  be 
reused  as  well  as  be  more  relevant  to  the  library.  We  hope 
that  through  this  process  that  software  developers  will  be 
able  to  use  our  correct  assertions  in  the  library  for  their 
own  use  and  reuse,  lessening  their  burden  and  reducing 
specification  errors  in  the  software. 

B .  STATECHART  ASSERTIONS 

The  libraries  are  built  through  the  use  of  "UML 
statechart  based  temporal  assertions  for  formal 
specif icat ions . The  UML  statecharts  are  developed  from 
both  the  research  efforts  of  Harel,  who  first  proposed  the 
use  of  statechart  diagrams  as  a  visual  approach  to  modeling 
the  behavior  of  complex  reactive  system,  and  Drusinsky  who 
both  increased  and  extended  the  use  of  statechart  diagrams 
to  specify  formal  assertions.  Drusinsky  was  able  to  extend 
the  use  of  statecharts  as  formal  assertions  for  temporal 
behavior  with  "the  inclusion  of  a  built-in  Boolean  flag 
bSuccess  and  a  corresponding  isSuccess  method  which 
specifies  the  Boolean  status  of  the  assertion  true  if  the 


D.  Harel.  Statecharts:  A  visual  approach  for  complex  systems. 
Science  of  Computer  Programming,  vol.8,  no. 3.  (1987) :  231-274. 
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assertion  succeeds  and  false  otherwise . The  statechart 
assertion  indicates  that  "formalism  is  supported  by 
StateRover,  a  design  entry,  code  generation,  and  visual 
debug  animation  tool  for  UML  statecharts  combined  with 
f lowchart s  .  Assertion  statecharts  can  be  nondeterminist ic 
and  deterministic  depending  on  the  needs  and  wants  of  the 
developer  and  modeler.  For  example,  the  developer  might  want 
a  nondetermist ic  statechart  if  there  are  nested  requirements 
which  can  be  more  difficult  to  write  and  less  readable  in  a 
deterministic  solution.  Or  alternatively  if  the  assertion 
needs  to  be  active  in  runtime,  then  a  deterministic 
statechart  might  be  a  better  solution  because  of  the 
overhead  incurred  in  the  nondeterminist ic  statechart  at 
runtime . 


Finally,  it  is  important  to  understand  the  proper  use 
of  a  statechart  assertion.  Remember  that  the  assertion  uses 
the  "built-in  Boolean  variable  name  bSuccess,  and  a 
corresponding  method  called  isSuccessO,  both  automatically 
created  by  the  code  generator"^^  to  make  a  statement  about 
the  assertion's  correctness.  The  default  settings  of  the 
assertion  statechart  variable  bSuccess  is  set  to  true.  To 
appropriately  test  success  and  failure,  the  modeler  needs  to 


^^2  D.  Drusinsky.  Modeling  and  verification  using  UML  statecharts  -  a 
working  guide  to  reactive  system  design,  runtime  monitoring  and 
execution-based  model  checking.  Elsevier  Inc.,  2006. 

D.  Drusinsky,  M.  Shing,  K.  Demir,  "Creation  and  validation  of 
embedded  assertion  statecharts",  Proc.  15^^  IEEE  International  Workshop 
in  Rapid  System  Prototyping,  Greece  (June  14-16,  2006) :  17-23 

D.  Drusinsky.  Modeling  and  verification  using  UML  statecharts  -  a 
working  guide  to  reactive  system  design,  runtime  monitoring  and 
execution-based  model  checking.  Elsevier  Inc.,  2006. 
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ensure  that  the  assertion  enters  the  error  state  and  the  on- 
entry  action  assigns  bSuccess=f alse  when  the  assertion 
fails . 

C .  ASSERTION  VALIDATION 

Once  the  natural  language  has  been  translated  into  an 
assertion  the  assertion  must  then  be  validated.  The 
assumptions  in  the  statechart  must  be  tested  to  ensure  that 
the  statechart  assertion  correctly  represents  the  intended 
behavior  the  modeler  has  in  mind.  We  need  to  run  validation 
test  scenarios  against  the  statechart  assertion. 

In  each  case  the  validation  test  suite  resolved  the 
ambiguities  of  the  natural  language  specification.  The  tests 
were  meaningful  in  that  they  ensured  each  assertion  were 
distinguishable  from  each  other.  The  assertions  were  tested 
and  we  did  find  that,  when  we  tested  them,  we  had  to 
disambiguate  the  natural  language  ourselves  to  ensure  that 
we  truly  understood  what  we  were  describing. 

The  two  kinds  of  errors  that  are  commonly  found  were 
"implementation  errors  resulting  from  mistakes  in  the 
statechart  assertion,  and  errors  or  ambiguities  in  the 
natural  language  statement."  In  the  first  case,  the 

statechart  behavior  does  not  match  the  modeler' s  intended 
behavior.  The  second  case  was  more  difficult  because  it 
depended  how  we  as  individuals  understood  the  natural 
language  statement  and  how  we  as  individuals  clarified  the 

T.  Otani,  D.  Drusinsky,  et.al.  "Validating  UML  statechart  based 
assertions  libraries  for  improved  reliability  and  assurance." 

Proceedings  of  the  Second  International  Conference  on  Secure  System 
Integration  and  Reliability  Improvement  (SSIRI  2008),  Yokohama,  Japan 
(July  14-17,  2008):  47-51. 

Ibid. 
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ambiguities.  It  was  by  running  the  test  scenarios  that  we 
were  able  to  identify  these  errors  and  modify  our 
assumptions  and  assertions  accordingly  in  order  to  correct 
the  assertions. 

Otani  et  al.^®  revealed  that  there  are  some  types  of 
patterns  that  must  be  part  of  every  validation  test-suite: 

•  Obvious  success.  We  expect  that  the  statechart 

assertion  being  validated  to  succeed  on  such  a  test. 

•  Obvious  failure.  We  expect  that  the  statechart 

assertion  being  validated  to  be  violated  on  such  a 
test . 

•  Event  repetitions.  We  create  event  repetitions  and 

assure  that  the  assertion,  if  applicable,  is  not 
written  in  a  manner  that  only  observes  the  first 
occurrence  of  a  triggering  event  P  in  a  sequence 
of  P's. 

•  Multiple  time  intervals.  If  the  assertion  requires 

it,  we  check  that  it  handles  multiple  time  intervals 
or  scenarios.  By  using  this  validation  test 

pattern  we  assure  that  an  assertion  is  not  written 
in  a  manner  that  observes  only  a  single  time 

interval . 

•  Overlapping  time  intervals.  If  the  assertion 

requires  it,  we  check  that  the  assertion  can  handle 
overlapping  time  intervals  within  a  scenario. 

Once  the  types  of  patterns  were  clarified  we  then 
designed  our  test  suite  to  adhere  to  the  above  categories, 
combining  them  if  suitable  and  ensured  that  there  were  an 
appropriate  number  of  tests  per  test  suite  that  would 
validate  the  assertion. 


T.  otani,  D.  Drusinsky,  et.al.  "Validating  UML  Statechart  Based 
Assertions  Libraries  for  Improved  Reliability  and  Assurance." 
Proceedings  of  the  Second  International  Conference  on  Secure  System 
Integration  and  Reliability  Improvement  (SSIRI  2008),  Yokohama,  Japan, 
(July  14-17,  2008):  47-51. 
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D. 


ASSERTION  SCENARIOS 


The  first  assertion  statement  that  we  described  is: 
Whenever  P  then  Q  within  T.  The  assertion  statechart  (shown 
in  Figure  3)  was  diagramed  as  follows: 


Whenever  P  then  Q  within  T 


‘\  Q[]/ 


static  final  int  T  =  30; 
TRTimeoutSimulatedTime  timer  =  tiew 
TRTimeoutSimulatedTime(  T,  this); 


\p[y 


Qlnit 

Op 

M - 

On-Entry /timer .  restartO; 

\Q[V 

Ni  timeoutFire[]/ 


t 

Q  Error 

On-Entry /bSuccess  =  false; 


Figure  3.  Whenever  P  then  Q  within  T 


Our  interpretation  of  the  assertion  statement  above  is: 

if  P  occurs  (timer  is  reset  at  every  P)  then  the  event  Q 

will  eventually  occur  within  the  time  interval.  The  built  in 

event,  t  imeoutFire  ( )  ,  fires  after  30  sec.  In  case  of  a  P 

repetition  before  a  Q  the  30  second  duration  will  be 

measured  from  the  first  p.  We  used  the  following  patterns  to 
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correctly  disambiguate  the  natural  language  and  ensure  that 


the  assertion 
language  as  we 
in  the  chapter 


statement  accurately  reflects  the  natural 
identified  and  desired.  As  described  earlier 
we  covered  all  appropriate  testing  patterns. 


Obvious  success: 


► 


P;  incrTime (25) ;  Q; 
We  expected  this  test  to 
confirmed . 


incrTime (6)  (timeout  has  occurred)  . 
be  a  success.  Our  expectation  was 


► 


Q;  incrTime (31)  (timeout  has  occurred) .  We  expected 
this  test  to  be  a  success  because  we  are  testing  for 
violations  of  the  assertion  and  Q  by  itself  does  not  violate 
the  assertion.  Our  expectation  was  confirmed. 

Obvious  failure: 

P 

30 

^ - ► 

P;  incrTime (31)  (timeout  has  occurred) .  We  expected 
this  test  to  fail  because  it  did  not  meet  the  constraints  of 
the  assertion.  Our  expectation  was  confirmed. 
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Overlapping  time  intervals: 


In  this  test  and  the  next  test  we  ensure  that  the 
assertion  observes  more  than  the  first  P  in  a  sequence  of 


P's. 

P  P  Q 


•4 

30 

i  30 

1 

1 _ : 

^ ^ 

P;  incrTime  (15) ;  P;  incrTime  ( 5 ) ;  Q;  incrT 
(timeout  has  occurred) .  Our  goal  in  this  test  was  to 
that  the  assertion  could  handle  overlapping  time  int 
We  expected  success.  Our  expectation  was  confirmed. 


ime  (2  6) 
ensure 
ervals  . 


P  Q  P  Q 


•4 - 

- ^  ^ 

30 

< - 1 - 30 

1 

1 _ : 

L-3 

i _ ^ ^ _ ] 

r _ ^ 

P;  incrTime ( 5 ) ;  Q;  P; 
occurred);  Q.  Our  goal  in  this 
time  intervals  for  an  expected 
meet  the  constraints  of  the  as 
confirmed . 


incrTime (31)  (timeout  has 
test  was  to  test  overlapping 
failure  as  this  test  does  not 
sertion.  Our  expectation  was 


Multiple  Intervals: 

We  tested  for  multiple  intervals  in  this  test  and  the 
next  to  ensure  that  the  assertion  would  observe  more  than  a 
single  time  interval. 
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P  Q 


P  Q 


30 


LS. 


± 


30 


P;  incrTime  ( 1 0 ) ;  Q;  incrTime  (20) ;  P;  incrTime  ( 1 0 ) ;  Q; 
incrTime(21)  (timeout  has  occurred) .  We  expected  success 
because  it  meets  the  requirements  of  the  assertion.  We  set 
bSuccess  =  true.  Our  expectation  was  confirmed. 


P  Q  P 


30 


LS. 


30 


P;  incrTime ( 1 0 )  ;  Q;  incrTime (20)  ;  P;  incrTime (31) 
(timeout  has  occurred) .  We  tested  for  multiple  intervals 
expecting  failure  because  of  the  constraints  of  the 
assertion.  Our  expectation  was  confirmed. 


The  second  core  assertion  statement  that  we  described 
was:  Whenever  P  then  no  Q  within  T.  The  assertion  statechart 
(as  shown  in  Figure  4)  was  diagramed  as  follows: 
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whenever  P  then  no  Q  within  T  static  final  int  T  =  30; 

TRTimeoutSimulatedTime  tinner  =  new 
TRTimeoutSimulatedTime(  T,  this); 


Figure  4 .  Whenever  P  then  no  Q  within  T 

Our  interpretation  of  the  assertion  is  if  P,  then 
within  the  time  interval  for  P  no  Q  will  appear.  The  built 
in  event,  t imeoutFire ( )  ,  fires  after  30  sec.  A  P  repetition 
would  reset  the  timer.  We  used  the  following  patterns  to 
disambiguate  the  assertion  statement. 


Obvious  success: 


P;  incrTime (31)  (timeout  has  occurred) .  We  expected 
this  test  to  be  a  success  because  no  Q  occurred  which  meets 
the  requirements  of  our  assertion.  Our  expectation  was 
confirmed . 

Q 

30 

^ - ► 


Q;  incrTime (31)  (timeout  has  occurred) .  We  expected 
this  test  to  be  a  success  because  we  are  testing  for 
violations  of  the  assertion  and  Q  by  itself  does  not  violate 
the  assertion.  Our  expectation  was  confirmed. 


Obvious  failure: 


P  Q 


This 

the 

conf 


30 


- * - - ► 

P;  incrTime (25) ;  Q;  i 
test  was  expected  to 
requirements  of  the 
irmed . 


ncrTime(6)  (timeout  has  occurred), 
be  a  failure  because  it  violates 
assertion.  Our  expectation  was 


Overlapping  time  intervals: 


In  this  test 
assertion  observes 
Ps  . 


and  the  next 
more  than  the 


test  we  ensure  that  the 
first  P  in  a  sequence  of 
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P  P  Q 


P;  incrTime (15) ;  P;  incrTime  (5);  Q;  incrTime(26) 
(timeout  has  occurred) .  Our  goal  in  this  test  was  to  ensure 
that  the  assertion  could  handle  overlapping  time  intervals. 
The  test  was  expected  to  be  a  failure.  Our  expectation  was 
confirmed . 


P  P  Q 


P;  incrTime ( 1 0 ) ;  P;  incrTime (31) (timeout  has  occurred); 
Q.  The  test  was  expected  to  be  a  success  because  Q  was  not 
injected  during  the  P  intervals.  Our  expectation  was 
confirmed . 

Multiple  Intervals: 

We  tested  for  multiple  intervals  in  this  test  to  ensure 
that  the  assertion  would  observe  more  than  a  single  time 
interval . 


P  P  Q 


P;  incrTime (30)  ;  P;  incrTime ( 15)  ;  Q;  incrTime (16) 
(timeout  has  occurred)  .  This  test  was  expected  to  be  a 
failure  because  it  does  not  meet  the  constraints  of  the 
assertion.  Our  expectation  was  confirmed. 

4  6 


E.  CONCLUSION 

When  we  first  defined  the  assertions  in  natural 
language  we  discovered  that  almost  all  assertions  can  be 
ambiguous  and  difficult  to  define  at  first.  The  natural 
language  statements  meant  different  things  to  different 
people.  "If  P  then  Q  within  T"  could  mean  an  interval  T 
measured  from  the  first  or  the  last  P  depending  on  how  it 
was  defined  and  what  the  software  developers  wants  to  test. 
We  disambiguated  each  assertion  according  to  the  most 
general  and  useful  definition;  this  meant  that  in  most  cases 
the  assertion  would  be  general  and  not  specific  so  as  to  be 
more  useful.  There  was  additional  difficulty  as  can  be 
expected  with  any  new  system  as  StateRover  is  still  in 
development.  But  we  were  able  to  succeed  after  several 
restarts  and  debugging  help.  Finally,  during  our 
disambiguation  period  we  fell  victim  to  the  statechart 
default  which  is  bSuccess  =  true.  During  the  testing  period 
we  expected  one  result  and  received  something  completely 
different.  This  led  us  to  additional  testing  and 
clarification  of  the  assertions  and  we  had  to  ensure  every 
time  that  the  assertion  test  was  not  successful  because  the 
bSuccess  flag  was  set  to  true,  but  rather  because  the  test 
was  actually  correct. 

This  process  is  incredibly  interesting  and  requires 
clarity  of  thinking  as  well  as  the  ability  to  break  down 
natural  language.  It  is  not  simple  but  the  process  invokes 
greater  understanding  of  the  validation  process  and  the 
validity  of  the  assertion  library.  We  feel  that  these 
assertion  statements  can  be  built  upon  and  reused  for  the 
benefit  of  validation  purposes. 
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Additional  assertion  statecharts  and  validation  test 
suites  that  we  defined  and  tested  can  be  found  in  Appendix 
A.  A  final  assertion  statechart  and  validation  test  suite 
that  has  merit  but  is  not  as  valuable  as  previous  mentioned 
assertions  can  be  found  in  Appendix  B. 
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V.  CONCLUSION 


A.  SUMMARY  AND  CONTRIBUTIONS 

Software  has  become  a  vital  part  of  our  everyday  lives. 
Whether  we  refer  to  our  military  systems,  medical  systems, 
or  our  financial  systems,  software  is  a  part  of  them  and  has 
become  something  that  we  now  depend  on.  In  our  thesis  we 
concentrate  on  requirements  and  their  formal  specification, 
and  we  discuss  a  method  to  reduce  specification  errors.  We 
strive  to  find  a  better  technique  to  answer  the  question 
"Are  we  building  the  right  product?"  Validation  presents  a 
means  of  assuring  that  software  satisfies  the  user' s 
requirements.  It  is  viewed  as  a  way  of  saving  time  and  money 
that  could  otherwise  be  wasted  if  a  product  design  is  not 
built  correctly  and  rework  needs  to  be  conducted.  A  problem 
that  can  exist  when  conducting  validation  is  not  conducting 
validation  early  enough  in  the  design  process.  Often  the 
user' s  requirements  are  reviewed  and  specifications  are 
developed.  The  product  design  is  then  built  according  to  the 
specifications.  Once  the  product  design  is  built  validation 
is  conducted  by  comparing  the  resultant  product  with  the 
original  requirements.  As  software  partition  of  systems 
continues  to  grow  and  become  more  complex  we  assert  that 
validating  a  product  after  it  is  developed  is  too  late  in 
the  process.  At  that  stage  the  amount  of  time  and  cost  of 
rework  that  may  need  to  be  performed  is  too  large. 
Validation  needs  to  begin  earlier  in  the  design  process  by 
ensuring  the  specifications  themselves  are  correct  and 
consistent . 
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At  present  several  ways  to  conduct  validation  exist. 
Some  guidance  that  is  provided  actually  describes 
verification  when  referring  to  validation  by  having  the 
product  design  compared  to  the  specifications  for  the 
project.  Others  suggest  what  we  have  already  mentioned  and 
that  is  to  check  the  final  product  against  the  user 
requirements.  To  conduct  validation  we  introduced  a  process 
of  developing  and  validating  temporal  formal  specifications 
in  the  form  of  statechart  assertions.  Included  in  our  work 
are  validation  test  scenarios  intended  to  ensure 
specifications  are  in  fact  correct  prior  to  moving  forward 
with  a  project.  The  goal  is  to  make  available  multiple 
libraries  of  pre-vetted  assertions  to  facilitate  validation. 

This  research  described  the  attributes  of  IV&V  as  well 
as  software  reuse  and  explores  a  concept  of  combining  the 
process  of  validation  and  reuse  in  an  attempt  to  yield  a 
repeatable  validation  technique.  Sample  requirements  were 
identified  and  then  formal  specifications  in  the  form  of 
statechart  assertions  were  created  to  capture  the 
requirements.  Testing  scenarios  were  then  developed  to 
determine  if  the  statechart  assertions  were  accurate  and 
consistent  with  the  original  requirements.  Once  these 
assertions  are  proven  to  be  accurate  they  can  be  stored  in  a 
library  for  future  reuse.  Our  intentions  are  to  ensure  that 
specifications  used  to  build  a  product  are  validated  prior 
to  time  and  money  going  into  building  the  final  product.  By 
using  an  assertion  repository  filled  with  correct  assertions 
to  build  the  specifications  for  a  design,  the  engineer  can 
be  sure  that  the  specifications  used  to  build  the  final 
product  are  correct.  If  errors  are  found  in  the 


specifications  the  engineer  can  go  back  and  find  out  where 
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the  error  is  coming  from.  This  would  be  faster  and  cheaper 
than  correcting  software  that  has  already  been  developed  in 
accordance  with  incorrect  specifications. 

B .  FUTURE  WORK 

The  goal  in  both  the  DoD  and  the  software  industry  is 
to  produce  software  that  is  cost  effective,  reliable, 
maintainable,  and  above  all  usable.  The  current  guidance  on 
verification  and  validation  that  exists  does  not  provide  a 
technique  to  show  engineers  how  to  create  software  that 
possesses  these  attributes.  The  guidance  that  does  exist 
leaves  software  engineers  to  develop  their  own  verification 
and  validation  methods. 

The  amount  of  work  that  could  be  conducted  in  the 
software  industry  to  ensure  reliable  software  is  being 
produced  is  abundant.  We  have  established  an  approach  for 
developing  and  validating  statechart  assertions  as  a  road 
map  to  produce  reliable  software.  One  avenue  of  future  work 
would  be  to  further  expand  this  approach  by  developing 
additional  assertions  that  apply  to  a  specific  domain.  For 
example,  select  a  domain  of  interest  such  as  theatre 
ballistic  missile  defense.  Then,  determine  requirements  that 
exist  within  that  domain.  Once  the  requirements  for  the 
specific  domain  are  understood,  translate  the  natural 
language  of  the  requirements  into  assertions  as  we  did  in 
chapter  IV.  Then  validate  the  assertions  through  the  use  of 
test  cases  to  ensure  that  the  statechart  assertions 
correctly  represent  the  intended  behavior  the  modeler  has  in 
mind . 
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Another  avenue  of  future  work  would  be  to  create  a 
library  to  store  the  assertions  in.  When  creating  the 
library  the  developer  will  need  to  consider  the  size  of  the 

library  and  how  many  assertions  will  be  placed  in  it.  The 

developer  will  need  to  decide  if  several  libraries  are  to  be 
developed  to  categorize  the  different  assertions  or  if  the 
assertions  will  be  organized  within  one  library  in  a  manner 
that  will  be  easy  to  search.  Once  the  organization  of  the 
library  is  decided  information  retrieval  will  need  to  be 

focused  on.  How  will  assertions  be  retrieved  or  called  from 

the  library?  What  will  be  the  best  interface  to  facilitate 
information  retrieval  and  the  use  of  the  assertions?  The 
goal  should  be  to  find  an  acceptable  interface  that  does  not 
cause  errors  of  its  own.  Another  area  to  look  at  is  the 
adaptation  of  the  assertions  to  a  library  environment.  Do 
they  perform  as  expected?  One  goal  the  developer  should  seek 
is  to  automate  the  processes  of  organizing,  retrieving 
information  from,  and  interfacing  the  libraries  as  much  as 
possible  in  an  attempt  to  reduce  errors. 

Finally,  once  a  library  is  developed,  a  future  project 

could  focus  on  how  to  best  maintain  that  library  to 

facilitate  future  use.  One  item  to  consider  is  if  certain 

assertions  are  used  more  frequently  than  others.  In  this 

case  the  developer  would  want  to  set  up  the  library  in  a  way 

that  the  frequently  used  assertions  can  be  searched  before 

the  rest  of  the  library  is  searched.  A  way  to  enable  this 

would  be  to  maintain  a  count  of  how  often  each  assertion  is 

used.  Also  if  an  assertion  is  proven  not  to  be  used,  a  way 

to  comment  the  assertion  out  in  the  library  to  eliminate  it 

from  future  searches  may  prove  useful.  Doing  this  may  be  a 

way  to  enable  faster  searches  thereby  saving  time  in  the 
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development  process.  By  commenting  the  assertion  out  rather 
than  removing  it  from  the  library  it  can  still  be  included 
in  future  searches  if  it  is  decided  that  it  is  needed. 
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APPENDIX:  ADDITIONAL  ASSERTION  DIAGRAMS  AND  TEST 

SUITES 


A.  ADDITIONAL  ASSERTION  DIAGRAMS  BOUNDED  BY  TIME 


1.  Whenever  P  Then  Less  Than  N  Qs  Within  T 


Whenever  P  then  less  than  N  Q's  within  T 


P 


30 


± 


► 


P;  incrTime(31)  (timeout  has  occurred) .  We  expected  an 
obvious  success.  Our  expectation  was  confirmed. 


P  Q 


30 

< - 

- > 

3 

3 

^ 

P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  an  obvious  success.  Our  expectation  was 
confirmed . 
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P  Q  Q 


30 

< - 

— 

_ 3 

: _ 3 

: _ 3 

i ^ ^ 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime(21) 
(timeout  has  occurred) .  We  expected  failure  since  we  set  N 
to  2.  Our  expectation  was  confirmed. 


P  Q  Q  Q 


30 


► 


P ;  incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ;  Q  ; 

incrTime (16) (timeout 

has 

occurred) . 

We 

expected  failure 

since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


P  Q  P  Q  Q 


30 

l _ 3 

1 _ Ls 

1 _ 3 

: _ 3 

1  3C 

P;  incrTime ( 5 ) ;  Q;  incrTime (26) (timeout  has  occurred); 
P;  incrTime ( 5 ) ;  Q;  incrTime (5)  Q;  incrTime (21)  (timeout  has 
occurred) .  We  tested  for  multiple  intervals  in  this  test  to 
ensure  that  the  assertion  would  observe  more  than  a  single 
time  interval.  We  expected  failure  in  the  second  interval 
since  we  set  N  to  2 .  Our  expectation  was  confirmed. 
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2. 


Whenever  P  Then  Less  Than  or  Equal  to  N  Qs  Within 
T 


P 

30 


— * - - ► 

P;  incrTime(31)  (timeout  has  occurred) .  We  expected  an 
obvious  success.  Our  expectation  was  confirmed. 

P  Q 


30 

< - 

- 

3 

3 

^ ^ 

P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  an  obvious  success.  Our  expectation  was 
confirmed . 


P  Q  Q 


30 

< - 

- ^ 

3 

3 

; 3 

^ ^ 
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P;  incrTime ( 5 ) ;  Q;  incrTime(5);  Q;  incrTime(21) 
(timeout  has  occurred) .  We  expected  an  obvious  success.  Our 
expectation  was  confirmed. 


P  Q  Q  Q 


P ;  incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ;  Q  ; 

incrTime (16) (timeout 

has 

occurred) . 

We 

expected  obvious 

failure  since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


P;  incrTime (5);  Q;  incrTime (26) (timeout  has  occurred); 
P;  incrTime  ( 5 ) ;  Q;  incrTime  (5)  Q;  incrTime  ( 5 ) ;  Q; 
incrTime  (16)  (timeout  has  occurred).  We  tested  for  multiple 
intervals  in  this  test  to  ensure  that  the  assertion  would 
observe  more  than  a  single  time  interval.  We  expected 
failure  in  the  second  interval  since  we  set  N  to  2.  Our 
expectation  was  confirmed. 
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3. 


Whenever  P  Then  Equal  to  N  Qs  Within  T 


Whenever  P  then  equal  to  N  Q's  within  T 


P 


30 


► 


P;  incrTime(31)  (timeout  has  occurred) .  We  expected 
obvious  failure  since  we  set  N  to  2 .  Our  expectation  was 
confirmed . 


P  Q 


30 

4 - 

- 

2 

: 2 

;  ^ 

P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  obvious  failure  since  we  set  N  to  2.  Our 
expectation  was  confirmed. 
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P  Q  Q 


30 

< - 

— 

_ 3 

: _ 3 

: _ 3 

i ^ ^ 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime(21) 
(timeout  has  occurred) .  We  expected  obvious  success  since  we 
set  N  to  2 .  Our  expectation  was  confirmed. 


P  Q  Q  Q 


30 


► 


P ;  incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ;  Q  ; 

incrTime (16) (timeout 

has 

occurred) . 

We 

expected  obvious 

failure  since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


P  Q  Q  P  Q  Q  Q 


30 

l _ 3 

1 _ 3 

1 _ Ls 

L-3 

L-1 

1  3C 

i _ 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime (21) (timeout 
has  occurred);  P;  incrTime ( 5 ) ;  Q;  incrTime (5)  Q; 
incrTime  ( 5 ) ;  Q;  incrTime  (16)  (timeout  has  occurred).  We 
tested  for  multiple  intervals  in  this  test  to  ensure  that 
the  assertion  would  observe  more  than  a  single  time 
interval.  We  expected  failure.  Our  expectation  was 
confirmed . 
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4. 


Whenever  P  Then  Greater  Than  or  Equal  to  N  Qs 
Within  T 


timeoutFire[l/'  __ 

On-Entry/bSuccess  =  false; 


^Whenever  P  then  there  must  be  greater  than  or  equal  to  N  Q's  within  T 


P 

30 

^ - ► 

P;  increment  time  to  31;  timeout.  We  expected  obvious 
failure  since  we  set  N  to  2 .  Our  expectation  was  confirmed. 

P  Q 


30 

4 - 

- 

3 

] 

P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  failure  since  we  set  N  to  2 .  Our  expectation  was 
confirmed . 


P  Q  Q 


30 

4 - 

- ► 

3 

: ] 

; ] 

^ ^ 
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P;  incrTime ( 5 ) ;  Q;  incrTime(5);  Q;  incrTime (21) 
(timeout  has  occurred) .  We  expected  success  since  we  set  N 
to  2.  Our  expectation  was  confirmed. 


P  Q  Q  Q 


30 


► 


P; 

incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ; 

Q; 

incrTime ( 5 ) ;  Q  ; 

incrTime 

(16)  (timeout 

has 

occurred) . 

We 

expected  success 

since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


P  Q  Q  P  Q 


•4— 

30 

•4— 

^  30 

1 

L-3 

L_! 

L_! 

L_! 

L _ ^ ^ 

P; 

incrTime ( 5 ) ;  Q; 

incrTime ( 5 ) ; 

Q; 

incrTime ( 5 

)  P; 

incrTime 

(5)  ; 

Q;  incrTime 

(26)  (timeout 

has 

occurred) . 

Our 

goal  in 

this 

test  was  to 

ensure  that 

the 

assertion 

could 

handle  overlapping  time  intervals  and  that  the  assertion 
observes  more  than  the  first  P  in  a  sequence  of  Ps.  The  test 
was  expected  to  be  a  failure  since  we  set  N  to  2  which 
violates  the  second  P.  Our  expectation  was  confirmed. 


P  Q  Q  P  Q 


30 

1  3( 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime (21) (timeout 
has  occurred);  P;  incrTime ( 5 ) ;  Q;  incrTime (26)  (timeout  has 
occurred) .  We  tested  for  multiple  intervals  in  this  test  to 
ensure  that  the  assertion  would  observe  more  than  a  single 
time  interval.  We  expected  failure  since  we  set  N  to  2 .  Our 
expectation  was  confirmed. 
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5. 


Whenever  P  Then  Greater  Than  N  Qs  Within  T 


ht  N  =  2; 
htcnt; 


Whenever  P  then  Greater  than  N  Q 's  within  T 


p 


30 


± 


P;  incrTime(31)  (timeout  has  occurred)  .  We  expected 
failure  since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


P  Q 


30 

4 - 

- 

3 

] 

P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  failure  since  we  set  N  to  2 .  Our  expectation  was 
confirmed . 
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P  Q  Q 


P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime(21) 
(timeout  has  occurred) .  We  expected  failure  since  we  set  N 
to  2.  Our  expectation  was  confirmed. 


P  Q  Q  Q 


30 


► 


P;  incrTime  ( 5 ) ;  Q;  incrTime  ( 5 ) ;  Q;  incrTime  (5);  Q; 
incrTime  (31)  (timeout  has  occurred)  .  We  expected  success 
since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


P  Q  Q  Q  P  Q 


►I 

30 

<r— 

►^30 

1 

L-3 

L-] 

L-] 

L-] 

L _ ] 

i _ ^ ^ 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime (5)  Q; 
incrTime  (5);  P;  incrTime  ( 1 0 ) ;  Q;  incrTime (21)  (timeout  has 
occurred)  .  Our  goal  in  this  test  was  to  ensure  that  the 
assertion  could  handle  overlapping  time  intervals  and  that 
the  assertion  observes  more  than  the  first  P  in  a  sequence 
of  P's.  The  test  was  expected  to  be  a  failure  since  we  set  N 
to  2.  Our  expectation  was  confirmed. 
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P  Q  Q  Q  P  Q 


30 

1 

r  ^ 

r  ^ 

r  ^ 

r  \  1 

r  ^ 

1  3C 

P; 

incrTime  ( 5 ) ; 

Q; 

incrTime  ( 5 ) ;  Q; 

incrTime ( 5 ) ; 

Q; 

incrTime 

(16) (timeout 

has 

occurred) ;  P ; 

incrTime ( 5 ) ; 

Q; 

incrTime (2 6)  (timeout  has  occurred) .  We  tested  for  multiple 
intervals  in  this  test  to  ensure  that  the  assertion  would 
observe  more  than  a  single  time  interval.  We  expected 
failure  since  we  set  N  to  2 .  Our  expectation  was  confirmed. 


6 .  Whenever  P  Then  Q  and  R  Within  T 


,  ,  .  . , .  static  final  int  T  =  30; 

Whenever  P  then  Q  and  R  within  T  TRTimeoutSimulatedTime  timer  =  new 

TRTimeoutSimulatedTime(T,  this); 


\  R[]/ 


\Q[]/ 
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p 


30 


P;  incrTime(31)  (timeout  has  occurred) .  We  expected 
failure.  Our  expectation  was  confirmed. 


Q 


Q;  incrTime(31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 


R 


R;  incrTime(31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 


P  Q 


P;  incrTime ( 5 ) ;  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  failure.  Our  expectation  was  confirmed. 
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P  R 


P;  incrTime ( 5 ) ;  R;  incrTime (2 6)  (timeout  has  occurred) . 
We  expected  failure.  Our  expectation  was  confirmed. 


P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  R;  incrTime (21) (timeout 
has  occurred).  We  expected  success.  Our  expectation  was 
confirmed . 


P  R  Q 


P;  incrTime ( 5 ) ;  R;  incrTime ( 5 ) ;  Q;  incrTime (21) (timeout 
has  occurred).  We  expected  success.  Our  expectation  was 
confirmed . 


P  P  Q  R 


P;  incrTime  ( 5 ); P ;  incrTime  ( 5 ) ;  Q;  incrTime  (5);  R; 
incrTime  (26)  (timeout  has  occurred)  .  Our  goal  in  this  test 
and  the  next  was  to  ensure  that  the  assertion  could  handle 
overlapping  time  intervals  and  that  the  assertion  observes 
more  than  the  first  P  in  a  sequence  of  Ps.  The  test  was 
expected  to  be  a  success.  Our  expectation  was  confirmed. 
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P  Q  PR 


•4— 

---►i 

30 

<— 

i  30 

1 

L_! 

: _ 3 

L-3 

: _ ^ ^ ^ 

P;  incrTime ( 5 ) ; Q;  incrTime ( 1 0 )  P;  incrTime  (5);  R; 
incrTime(26) (timeout  has  occurred) .  The  test  was  expected  to 
be  a  failure.  Our  expectation  was  confirmed. 


P  Q  R  P  Q 


P;  incrTime (5);  Q;  incrTime ( 5 ) ;  R;  incrTime (21) (timeout 
has  occurred);  P;  incrTime (5);  Q;  incrTime (26)  (timeout  has 
occurred) .  We  expected  failure.  Our  expectation  was 
confirmed . 

In  this  assertion  statement  we  did  not  differentiate 
between  Q  or  R  coming  first,  our  intention  was  to  ensure 
that  the  combination  of  both  Q  and  R  regardless  of  order 
resulted  in  a  successful  test. 
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Whenever  P  Then  Q  or  R  Within  T 


7  . 


whenever  P  then  Q  or  R  within  T 

static  final  int  T  =  30; 
TRTimeoutSimulatedTime  tinner  =  new 

TRTimeoutSimulatedTime(T,  this); 

Error 

N4  timeoutFire[]/ 

- ► 

On-Entry/bSuccess  =  false; 

P 

30 


± 


► 


P;  incrTime(31)  (timeout  has  occurred) . 
failure.  Our  expectation  was  confirmed. 


Q 

'4 


30 


We  expected 


Q;  incrTime(31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 
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R 

30 


— * - - ► 

R;  incrTime(31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 

P  Q 


30 

< - 

- 

_ 3 

: _ 3 

: _ ^ ^ 

P ;  incrTime  ( 5 ) ; 
We  expected  success. 


Q;  incrTime (26)  (timeout  has  occurred). 
Our  expectation  was  confirmed. 


P  R 


30 

< - 

- > 

_ 3 

1 _ ] 

i _ ^ ^ 

P ;  incrTime  ( 5 ) ; 
We  expected  success. 


R;  incrTime (26)  (timeout  has  occurred). 
Our  expectation  was  confirmed. 


P  Q  R 


30 

< — 

- >1 

_ 3 

1 _ ] 

: _ ] 

i _ ^ ^ 

P;  incrTime (5);  Q;  incrTime ( 5 ) ;  R;  incrTime (21) (timeout 
has  occurred).  We  expected  success.  Our  expectation  was 
confirmed . 
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P  R  Q 


P;  incrTime ( 5 ) ;  R;  incrTime ( 5 ) ;  Q;  incrTime (21) (timeout 
has  occurred) .  We  expected  success.  Our  expectation  was 
confirmed . 


P  Q  PR 


P;  incrTime ( 5 ); Q;  incrTime (15) ;  P;  incrTime ( 5 ) ;  R; 
incrTime  (26)  (timeout  has  occurred)  .  Our  goal  in  this  test 
and  the  next  was  to  ensure  that  the  assertion  could  handle 
overlapping  time  intervals  and  that  the  assertion  observes 
more  than  the  first  P  in  a  sequence  of  P's.  The  test  was 
expected  to  be  a  success.  Our  expectation  was  confirmed. 


P  P  Q  R 


P;  incrTime ( 5 ); Q;  incrTime (10)  P;  incrTime (31)  (timeout 
has  occurred);  R.  The  test  was  expected  to  be  a  success. 
Our  expectation  was  confirmed. 
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P  Q  R 


P  Q 


P;  incrTime(5);  Q;  incrTime(31) (timeout  has  occurred) ; 
P;  incrTime(5);  Q;  incrTime(5);  R;  incrTime(21)  (timeout  has 
occurred) .  We  tested  for  multiple  intervals  in  this  test  and 
the  next  to  ensure  that  the  assertion  would  observe  more 
than  a  single  time  interval.  We  expected  success.  Our 
expectation  was  confirmed. 


8 .  Whenever  P  Then  Q  or  Rot  R  Within  T 


>- 


P; 

obvious 


incrTime(31)  (timeout  has  occurred), 
success.  Our  expectation  was  confirmed. 


We 


expected 
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Q;  incrTime  (31)  (timeout  has  occurred)  .  We  expected 


success.  Our  expectation  was  confirmed. 

R 

30 

< - 

— * - ^ - ► 

R;  incrTime (31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 

P  Q 


30 

< - 

- 

_ 3 

: _ 3 

: _ ^ ^ 

P ;  incrTime  ( 5 ) ; 
We  expected  success. 


Q;  incrTime (26)  (timeout  has  occurred). 
Our  expectation  was  confirmed. 


P  R 


30 

< - 

- > 

_ 3 

i _ ] 

i _  ^ 

P;  incrTime ( 5 ) ;  R;  incrTime (26)  (timeout  has  occurred). 
We  expected  failure.  Our  expectation  was  confirmed. 


P  Q  R 


P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  R;  incrTime(21) (timeout 
has  occurred) .  We  expected  failure.  Our  expectation  was 
confirmed . 


P  P  Q  R 


P;  incrTime ( 15 ); P ;  incrTime ( 5 ) ;  Q;  incrTime (26) (timeout 
has  occurred) ;  R.  Our  goal  in  this  test  and  the  next  was  to 
ensure  that  the  assertion  could  handle  overlapping  time 
intervals  and  that  the  assertion  observes  more  than  the 
first  P  in  a  sequence  of  P's.  The  test  was  expected  to  be  a 
success.  Our  expectation  was  confirmed. 


P  P  Q  R 


P;  incrTime ( 5 ); Q;  incrTime (10)  P;  incrTime (21) ;  R; 
incrTime (10)  (timeout  has  occurred) .  The  test  was  expected 
to  be  a  failure.  Our  expectation  was  confirmed. 
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30 


P  Q  PR 


2 Z  Z—L 


30 


P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred); 
P;  incrTime(5);  R;  incrTime(26)  (timeout  has  occurred).  We 
tested  for  multiple  intervals  in  this  test  to  ensure  that 
the  assertion  would  observe  more  than  a  single  time 
interval.  We  expected  failure.  Our  expectation  was 
confirmed . 


9 .  Whenever  P  Then  Q  and  Not  R  within  T 


whenever  P  then  Q  and  not  R  within  T 


TRTimeoutSimulatedTime  timer  =  new 
TRTimeoutSimulatedTime(30j  the); 
//timeoutFire  (transition) 


\p[y 

Qinit 


\P[K 


Op 

On-Entry/timer.restart(); 


\  R[]/ 


□  Success 
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p 


30 


P;  incrTime (31)  (timeout  has  occurred) .  We  expected 
obvious  failure.  Our  expectation  was  confirmed. 


Q 


Q;  incrTime (31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 


R 


R;  incrTime (31)  (timeout  has  occurred) .  We  expected 
success.  Our  expectation  was  confirmed. 


P  Q 


P;  incrTime (5);  Q;  incrTime (26)  (timeout  has  occurred). 
We  expected  success.  Our  expectation  was  confirmed. 
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P  R 


30 

< - 

- 

_ 3 

: _ 3 

: _ ^ ^ 

P;  incrTime ( 5 ) ;  R;  incrTime (2 6)  (timeout  has  occurred) . 
We  expected  failure.  Our  expectation  was  confirmed. 

P  Q  R 


30 

- 

_ 3 

: _ 3 

: _ 3 

: _ ^ ^ 

P;  incrTime (5);  Q;  incrTime ( 5 ) ;  R;  incrTime (21) (timeout 
has  occurred)  .  We  expected  failure.  Our  expectation  was 
confirmed . 


P  Q  Q 


30 

< - 

— >1 

_ 3 

1 _ 3 

: _ 3 

i _  ^ 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime (21) 
(timeout  has  occurred).  We  expected  success.  Our  expectation 
was  confirmed. 


P  Q  Q  P  Q 


— 

30 

< — 1 - 

30 

1 

L-3 

L-3 

L-3 

: _ ^ _ 3 

^ ^ 

P;  incrTime  ( 5 ) ;  Q;  incrTime  ( 5 ) ;  Q;  incrTime  ( 5 ) ;  P; 
incrTime  (20) ;  Q;  incrTime (15)  (timeout  has  occurred).  Our 
goal  in  this  test  was  to  ensure  that  the  assertion  could 
handle  overlapping  time  intervals  and  that  the  assertion 
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observes  more  than  the  first  P  in  a  sequence  of  P's.  The 
test  was  expected  to  be  a  success.  Our  expectation  was 
confirmed . 


P  Q  Q  P  Q 


30 

1  3( 

P;  incrTime ( 5 ) ;  Q;  incrTime ( 5 ) ;  Q;  incrTime(21) 
(timeout  has  occurred);  P;  incrTime ( 5 )  ;  Q;  incrTime (26) 
(timeout  has  occurred) .  We  tested  for  multiple  intervals  in 
this  test  to  ensure  that  the  assertion  would  observe  more 
than  a  single  time  interval.  We  expected  success.  Our 
expectation  was  confirmed. 

B.  ADDITIONAL  ASSERTION  DIAGRAMS  UNBOUNDED  BY  TIME 

We  felt  that  this  assertion  statement  should  be 
separated  from  the  other  assertion  statements  because  the 
time  is  unbounded.  It  still  has  merit  as  an  assertion 
statement  but  may  not  be  as  useful  as  the  other  assertions. 
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1. 


Whenever  P  Then  Not  Q  After  T 


whenever  P  then  not  Q  after  T 


TRTimeoutSimulatedTime  timer  =  new 
TRTlmeoutSimulatedTime(3Q,  this); 
//timeoutFire  (transition) 


(~~)lnit 

^  Op  a 

Success 

^ - ► 

On-Entry/ timer  .restartO; 

\  timeoutFire  [y 

\Q[]( 


\Q[]/ 


▼ 

Q  Error 

On-Entry /bSuccess  =  false; 


30 


P;  incrTime(31)  (timeout  has  occurred) .  We  expected 
obvious  success.  Our  expectation  was  confirmed. 


Q 


30 


Q;  incrTime(31)  (timeout  has  occurred) 
success.  Our  expectation  was  confirmed. 

P  Q 


We  expected 


30 

< - 

- > 

3 

3 

^ ^ 

P;  incrTime(5);  Q;  incrTime(26)  (timeout  has  occurred). 
We  expected  success.  Our  expectation  was  confirmed. 
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P;  incrTime(31)  (timeout  has  occurred) ;  Q  .  We  expected 
failure.  Our  expectation  was  confirmed. 


P;  incrTime  ( 1 0 ) ;  P;  incrTime ( 1 0 ) ;  Q; 

incrTime(21)  (timeout  has  occurred)  .  Our  goal  in  this  test 
was  to  ensure  that  the  assertion  could  handle  overlapping 
time  intervals  and  that  the  assertion  observes  more  than  the 
first  P  in  a  sequence  of  Ps.  We  expected  success.  Our 
expectation  was  confirmed. 


P;  incrTime (5);  P;  incrTime (31) (timeout  has  occurred); 
Q.  We  expected  failure.  Our  expectation  was  confirmed. 
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P  Q  P  Q 


P;  incrTime(5);  Q;  incrTime(26) (timeout  has  occurred) ; 
P;  incrTime(31) (timeout  has  occurred);  Q.  We  tested  for 
multiple  intervals  in  this  test  to  ensure  that  the  assertion 
would  observe  more  than  a  single  time  interval.  We  expected 
failure.  Our  expectation  was  confirmed. 
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